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(54) Fair queuing system with adaptive bandwidth redistribution 



(57) Apparatus for routing packets in a communica- 
tion network comprises a plurality (100) of per-connec- 
tion queues (20a-20i), each queue established for 
receiving packets from a respective source and tempo- 
rarily storing received packets before routing to a partic- 
ular destination; a weighted fair<|ueuing scheduler (40) 
for servicing packets from each of the plurality of per- 
connectton queues at guaranteed pre-al located rates; a 
sensing device for sensing a presence or absence of 
packets in queues, the absence of packets in queues 



indicating availability of excess bandwidth; and, a state 
dependent scheduler (200) for redistributing excess 
bandwidth upon sensing of queues absent packets, the 
state dependent scheduler (200) servicing those 
queues in accordance with a state variable correspond- 
ing to a performance property of the queues, wherein 
delay and isolation properties for routing packets of 
respective queues in weighted fair-queuing is pre- 
served. 
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Description 

•j . - h .. ...... 

Reld of the Invention J 

The" instant invention* relates generally to packet 1 5 
based comrriuriicatidn' systems and particularly to fair- : 
queuing systems implemented 'in routers 1 and switches 
i n a packet-based communication system. ■•■■■■«*- v. 

Background of the Invention 1 i* : - 1 to 

Much research has been devoted *td development - - 
of queuing systems for" packetbased communication • ' 
networks that emulates as closely as possible " ah ideal "'- ' 
"fluid flow" model, i'.ef/ where data packets' comrnuhi- is 
cated from multiple sources are considered to-be Wi-^- 1 - 
nitely divisible^ and multiple sources'may transmit their v 
data simultaneously, e.g., on a single physical commu- " :n 
nication link. Infinite divisibility' is hot' feasible in practice. ' • 
In packet networks, typically bhce a packet is transmit- 20 
ted over the link,' the whole' packet must be sent, i.e.; ft *• 
cannot be interrupted to 'transfer 'another* packet in 1 - 
between. As there exists a desl re to provide; Quality- of ' ,; 
Service "QoS M guarantees' in a padket network, there is ,r * 
required the implementation of traffic sch^dulihg rrieth- 25 
ods in the data packet switches or routers. The function 
of a scheduling method is to select, for each 'outgoing*' 
link of a switch, the packet to be transmitted^ the next 
cycle from the available packets belonging to the coin- : 
munication sessions sharing the output link. This selec-- r 30 
tion must be perfdrmed such that the QoS guaranteed" ' 
for the individual traffic sessions, e.g., upper bounds on ' 
maximum delay, are satisfied. Implementation of the 
method may be hardware or software, but because of A 
speed considerations, scheduling is usually imple- 35 
merited in hardware in ATM switches and high-speed 
routers. ~ * " 

Many different scheduling methods have been pro- :cs: 
posed to approximate the theoretical* scheduling disci- 
pline known as Generalized Processor Sharing (GPS) 40 
system, which is a t discipline defined with respect to the 
"fluid" model. Such a GPS would allow for tight control of 
the bandwidth allocated to each session communicating 
on a link. However, as packets transmitted by a session 
cannot be divided further, the data "from multiple 45 
sources must be interleaved only at packet boundaries. 
Thus, the GPS discipline cannot be implemented in 
practice in a packet : swrtched network. 

/.Servicing of separate queues by simple FIFO, : 
Round Robin, and' fair queuing techniques, and the like,- * so" 
ar,e weli-knowri.] However, "Weighted"' fair-queuing"* 
("WFQ"), schemes have been developed that closely ' 
approximate the fluid system. Particularly, A. Demers, 
S, Keshav, S. Shenker, in the reference "Analysis and* 
Simulation of a Fair Queuing Algorithm" Internetwork- 55 
ing: Research and Experience, pp, 3-26, vol. 1, 1990 
describe a fair* queuing scheme that emulates GPS by 
essentially simulating a fluid flow GPS system for refer- 



2 

ence^and basing packet scheduling, decisions on the 
order: of departures in the GPS system. In weighted fair ^ 
queuing, each traff ic session / sharing : the output -link « 
controlled by the scheduling method is assigned a value ... 
4>i corresponding to the reserved bandwidth of the ses- 
sion. The values 4>j are computed such that the reserved 
bandwidth of session / on the link is given by: 



$ - 1 

AJ 

. ;' , ' \ 

where the denominator computes the sum of the <|>j, val- 
ues for : alJ t u sessions sharing the link. ^ 

'^Particularly, as shown in, Rig. 1 , a VVFQ system 100 
is provided with a plurality of per-connection queues 
20a,...,20i; with each queue storing. packets in a differ- r 
ent portion of a shared memory 25 for.temporarily stor- : 
ing packets ^of~ information,: e.g., -input traffic from a 
source' device such as a data terminal. It is understood 
thaf there 'can be provided different types of queues for 
accommodating 1 - different *♦ types :of traffic,- ag., audio, , 

• video, data; etc; Additionally, provided Js a shaper . 
30a;.. j .i30i that forward packets from the queues tdthe 
Weighted . Fair Queueing Server with a rate exactly 
equal to the allocated. The Weighted Fair Queueing . 
scheduler assumes that a weight is associated with 
'each queue 20a, ;.,20i; respectively, so that the service 
offered by the scheduler to each one of these queues 
while they have packets waiting is always in proportion 
to the weights: For example, let us -assume that the-, 
capacity '(bandwidth) of the link C=10packets/sec. Let 

• us also assume that the scheduler is serving three 
queues; Q1 being accorded ■? a /weight . WQ 1^=20%, 
queque G2 being accorded a weight WQ2=30%, and 
queue Q3 being accorded a weight WQ3=50%. Then, if 
all queues'have packets waiting, then Ql v and Q2 will 
receive a guaranteed bandwidth of 2 and 3 packets/sec- 
ond 'respectively, and Q3 will: receive a guaranteed 
bandwidth of 5- packets/second* However, if, for exam- 
ple,' Q3 does' not- have any packets waiting, then the 
excess bandwidth is equal to 5 packets/second. In a 

: * WFQ system, this excess bandwidth; is redistributed in 
proportion to the associated weights of the. queues that 
have packets waiting. In the above example, when 
queue Q3 does not have packets waiting, the excess 
bandwidth wilt be distributed proportionally to, queues 
Q1 and G2 so, that they now receive instaneous band- 
width of 4 and 6 packets per second respectively. Each 
packet leaving its respective shaper 30 is forwarded, 
directly to a Rate Proportional Server 40 ("RRS"). which - 
may be any weighted fair queuing variation, thaUforr 

• wards the packets to output link 50. 
In such a WFQ scheme, beneficial properties exist 

such as end-td-erid delay guarantees, e.g., each packet 
is' guaranteed a certain rate for each packet flow in the 
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stream, and, the provisions-isolation between streams^ 
e.g., a 'misbehaving source will not effect the Jlow; of: 
other streams. Additionally, an added benefit is that 
when- there is uhderutilization of capacity, e.g., when 1 
flow is particularly bursty and there may be idle time, the 
WFQ system facilitates the redistribution of the unused 
bandwidth so as to preserve work-conservation prop- 
erty. Presently, the redistribution property of unused 
bandwidth capacity among the queues is done in a 
manner inherited from the- fluid-flow model, e.g., in 
accordance with the weight associated with the particu- 
lar queue. Thus, when the packet queues are idle, 
"excess" bandwidth is re-distributed to backlogged con- 
nections in proportion to their weights which are based 
on Idng-terrh requirements. • - ; .:?&'* ;*c f r -n e 

The drawback of GPS that all fair queuing syfetem^r i- 
inherit in their : cF6se* emula'tionrof. -GiPS.is that GPS 
severely restricts state-dependent ^bandwidth scaring.:. 
The only state-dependfency in GB&isiin the number of f; , 
backlogged connections. There is: no further latitude 
and sharing is determined by the guaranteed; rates , 
which are' set based on long term needs .oHhe^cpnreHSr;.. 
tions. This* fektriction on 'bandwidth sharing- is _ more;: 
stringent than that necessary to preserve a key of prop-; 
erty-of fair queuing.' the ability tovguarantee- worst cas£ . 
delay bounds for leaky bucket controlled traffic-sources,^ 
Consequently, there ismoneed for fair queuing'systems^ 
to emulate the possibly suboptimal excess bandwjdtlv 
sharing of GPS. • iu :. i.„-:;rt 

It would' thus be highlyrdesirabte to* provide, jn-a^ 
weighted fair queuing system emulating GPS. a irtethpaV 
of achieving redistribution of unused bandwidtjvin r a- 
state-dependent manner, i.e.; that reflects instantane- 
ous needs of the remaining backlogged trafficilpw^, t 
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case fairness measures. ?:n-' 
The LDF policy uses excess bandwidth to reduce 
the variance of the delay distribution which has the , . 
added benefit of reducing the playout buffer' size for' 
voice and video sources. Simulations with video traces 
and wjth voice.traffic shows that indeed this policy per- _ 
forms better than GPS .without any sacrifice pf worst 
case guarantees. Since the. deviation frorri, the maxi- 
mum allowed delay is not taken into account, flows with 
small delay bounds (like voipe), ge^^lmpst no excess 
bandwidth in the presence of flows with large delay 
bounds. Irjiroracisgnefs jn assigning ^eights can result 
in these fjpws^ experiencing ^Ipssejs much more than 
f lows with large delay boundV _ . . s , . 
- ^e.LTO'pp^ tp'minifnize^ packet losses by 
assigning excess bandwidth under th^ assumption that 
the tow whicl^ is likely to pverfloyv the quickest, hasjthe 7 
mostrinstantanepus J^ndwidfh n^ed. In doing so, it 
takes : intp account the' current deviation, of each flow, 
from its maximum ^ with a mix 

of GBB,; wic^and ^^-Jt^^^-^M v ? rv f 1 !f fe r erit 
delay bpur^sshqyyj^ 

classes as wgll .as! : reduces! thg variance pf delay for 
each class. The various features of novelty which char- '; 
acterize the, inversion, are point ed.dgt ., 

wit^artiqulari.ty in the claims.annexed to and form- 
ing^ part oUhf.' disclosure. For a better understanding 
of the the invjentipa jtSj operating advantages! and spe- 
cif ic objects attajned by its us^, reference should be had 
:to the.cfrawings'an^ in which there 

are illustrated and-described preferred embodiments of ' 
the invention. - ^ > . , ? 
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Summary of the Invention 



' The instant invention's a:modified approapr^Jp i^. 
weighted fairqueuingfimplementing>an„adaptivei§dist[i- r> 
bution scheme. In such a scheme*, each per connections . .40 
flow ^guaranteed its: specified Shane^of mei.linfcbapdrr;.- 
width with any excess bandwidth being. adaptivelyre^lis; 
tributed. 'The scheme enables preservation): of < fair; 
queuings ability to provide worst .case end tp-er^d delay * c ; 
bounds andthe schemes work* like fair. queuing when - 45 
there is no 'excess bandwidth:. The .excess bandwidth t: 
may be distributed according to -an different • .criteria.: Ps 
Examples of state dependent crrteria; are: V) Longest,-, 
delay first (LDF) that serves: the flow with current Long-,- 
est delay; 2) Least time to overflow (LTQ) that, serves tjjie 
flow with minimum> difference between maximurn ... 
allowed 1 delay and current delay; -3) Least time* to over r ^ 
flow with leaky. bucketso(LTO-LB) that serves tr^e flow . r ' 
which; would cause buffer loverf low first if worst case r 
arrivals happen. <■*- c ■■■ ■ ■ ..;r-'.--v,- 

< Advantageously, the adaptive bandwidth redistribu- 
tion scheme always provides the minimum guarantees 
required for each of the connections and satisfies worst 
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Brigf Description of Drawings , ? x 

RJgI 1. is a .g£?nerardata flow diagramillustratjng the 
features of the weighted fair queuing and a rate propor- 
tionaLSeryer . . ); . ; . 

rFIG..^illustrates thp^eighteS fair'qu^uing schedul- 
ing system of the invention. ' t '\ t * 

Detailed Description of the Invention 

\ Figure.^2 "is % (general block diagram illustrating the 
,, fair queuefng .method wjth adaptive excess' bandwidth 
redistribution. Jhe system consists of a set of queues 
10o! a s.haper:. device 30a,..3bi associated "with each 
queue! and Weighted fair Queueinq Server pr Rate 
Proportipna^Seryer 40 and a State Dependent Server' 
; (SD?) ,200. Packets are. temporarily stored iri the 
queues 20a,... 2Qi. A weight is assigned with each of 
these queues representing the portion of the output 
bandwidth that must be allocated to the packets arriving 
in-each of the queues. The shaper device is forwarding 
. packets from the.queues to the Weighted Fair Queueing ^ 
Server with a rate, exactly equal to the allocated. This 
scenario .enables, excess bandwidth to become availa- 
ble for redistribution.. In accordance with'the principles 
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of the invention, traffic from another source, i-e., another 
queue7 may be directed to the adaptive bandwidth redis- 
tribution mi^anismf which is hereafter referred to as 
state dependent scheduler 200. It shouW.be understood 
to skilled artisans that the., hardware architectures for 
implementation of the perconheetion queues, shaper 
devices, and schedulers may vary, without diverting or 
straying from the principles of the invention described 
herein. • ' •■'-* • . » 

As described above; the' shaper releases packets to 
the scheduler with a rate exactly equal to the allocated 
rate. The service offered by the shapers30a,..,30i tothe 
respective queue connections during an interval of 
time is denoted by Sj(x,t) in Figure 2. It is assumed that 
packets are transferred from the shaper to the sched- 
uler with infinite capacity. The service offered by the rate 
proportional scheduler 40 is denoted as R\ (x,t). 

Packets that have not become eligible for service 
remain in the corresponding connection queue in the 
shaper, while all the eligible packets wart for service in 
the RPS Scheduler queue 40. Service is always pro- 
vided from the RPS scheduler queues 40, as long as 
packets are available there. When all of the scheduler 
queues are empty, the state SDS 200 is invoked to 
select a packet from the queues 20a,...20i-for transmis- 
sion with the service offered to a connection not affect- 
ing the state of the shaper 30. As shown in Fig. 2, the 
service offered by the shaper 30 to the queue connec- 
tion T during an interval of time is denoted by Dj(x,t) 
and the service offered by the state dependent sched- 
uler 200 is denoted as Z x (x,t). 

The shaper device 30 can be implemented using a 
calendar queue that is a well known mechanism to any 
skilled artisans. A description of the calendar queue 
mechanism for shaping has been presented in D. Stili- 
adis and A. Verma: A General Methodology for Design- 
ing Scheduling and Shaping Algorithms, in Proceedings 
of IEEE INFOCOM *97. 

The Rate Proportional Server (Weighted Pair 
Queueing Server) 40 can be implemented using any 
known fair queueing mechanisms of which there are 
several known to those skilled in the art. Examples are 
D. Stiliadis and A. Varma: Traffic Scheduling System 
and Method for Packet-Switched Networks. U.S. Patent 
Application, Serial No, 08/634,904, filed April 15,1996. 

At some point of time, if it is determined that the are 
no packets in the RPS scheduler, then this is an indica- 
tion of free bandwidth. Thus, packets may be served by 
the SDS 200 based on a state dependent manner. By 
state dependent manner, it is meant that the decision is 
based on the current state of some variables related to 
the system. Two examples of state dependent decisions 
are Longest Delay First and Least Time to Overflow. 

In the Longest Delay First, the scheduler SDS 200 
selects among the queues 20a.. ..20i. the one that has 
the packet that will see the longest delay if this queue is 
served with a rate equal to guaranteed, To do that, the 
scheduler assigns to each queue a delay value that is 



equatto the size of the queue divided by the guaranteed 
rate. This value describes the delay that the last packet 
of this queue will see. The SDS 200 selects the queue 
with the maximum such value using any mechanism for 

5 selecting the maximum value among a set of numbers. 
The selection .mechanism is a well-known .method to 
any skilled artisan. 

;ln the Least Time to Overflow mechanism, the SDS 
scheduler 200 selects among the queues ,20a,. ...20i 

10 that queue which is likely to overflow in the shortest 
time. The mechanism assumes that a maximum size is 
associated with a queue. Every time a packet arrives or 
is serviced by a queue, ,a packet-to-overflow variable 
counts the number *of packets that can be added-to the 

75 - queue without exceeding its maximum capacity. A sec- 
ond variable, denoted as* time-to-overflow, indicates the 
expected time to overflow and is calculated by dividing 
the packet-to-overflow variable by the associated rate. 
The SDS 200 selects for transmission the queue with 

20 " the minimum time-to-overflow variable: The minimum is 
determined by any of the well known mechanisms to 
anyone skilled in the art. Although the above variables 
explain two methods for calculating state 

dependent variables, other similar methods may be 

25 used. Based on the state dependent variables; the 
schedUler:mechanism 200 will direct packets from .the 
selected'queues over separate link 75 where the packet 
will be multiplexed by multiplexer element 350 and for- 
warded to its next destination. 

30 - It should be understood that the property- of worst 
case fairness is also satisfied in the system oftfie inven- 
tion. In worst case fairness, the servicing of queues of 
different lengths at different allocated rates is inter- 
leaved such that a packet from a first queue is serviced 

35 ■ at time X^ and the packet of a second queue is serviced 
at a time t 2 rln the invention, the measure of worst ease 
fairness is satisfied, such that, the time interval t 2 -ti, in 
the worst "case, is less than or bounded by a value that 
is hot a function of the number of connections, but is a 

40 - ; function of the packet size of. the longest queue; e.g., 
the queue serviced at time Xy and the allocated rate of 
that longest queue. 

< Thus, the property of worst case fairness is satis- 
fied by the implementation of the SDS 200 in the 

45 weighted fair queuing system of the invention, * 

The foregoing merely illustrates the principles of the 
present invention. Those skilled in the art will be able to 
devise various modifications, which although not explic- 
itly described or shown herein, embody the principles of 

50 the invention and are thus within its spirit and scope. 

Claims ' , 

Apparatus for routing packets in a communication 
55 * network comprising: 

^ a plurality of per-connection queues, each 
queue established for receiving packets from a 
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3. 



respective source and temporarily • storing j. 

; ' received packets before routing to a particular. 

-■destination;' ^ J,: ' * 
'weighted fair queuing scheduling means serv- 
icing packets from each of said plurality of per- 
connection queues at guaranteed pre-allo- 
cated rates; 

- means for sensing a presence or . absence of 
: packets in queues, said absence of packets in 

queues indicating availability of excess band- 
width; and, • ■■ - 

- state* : dependent scheduling- means for redis-r ; 
Mributing excess bandwidth upon sensing of; 

queues absent said " packets, said e state. ! 
dependent scheduling' imeans.servicing those i 
: queues in accordance with a staterVariable corv; 
respondinjgto a performance property of satch 
queues, - • v u : * ' T ' v '~ "- 

wherein delay and isolation .properties 
for routing packets of . respective queues-is pre- 
served. * ■ - ' - r . • ' 1 r - 
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Apparatus for r routing packets in ar.corornurueatipn 
network- as claimed in Claim 1 t wherein a said per- 
formance property of: a queue includes the queue, ,25 
having' the greatest amount of packets, said corre-: 
: sponding state variable being longest queue first.- . : 



Apparatus for routing packets in a; communication- 
network as claimed in Claim 1, wherein a.said per- 
formance property of a queue includes that queue 
having a packet waiting to be serviced in the long-., 
est amount of time, said corresponding state varia^. 
ble being longest delay first. ^ : • ... .' 

* Apparatus for routing packets in-a communication ; 
network as claimed in Claim .1; wherein a said per-; f 

• formance property of a queue includes the queue- 
•* most likely to overflow buffer memory ^containing 

said queue, saidxorresponding<state variable being 
most likely to overflow buffer.- - . ■ 
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5. Apparatus for routing packets in a communication 
network as claimed in Claim .1, wherein a worst . 
case fairness measureior said connections is satis- - 45 

- fied. ' :r. ;• ■ 

6. > Method Grouting packets in a communication net- 

work comprising: :*r- *• . • : < \ 

establishing a plurality of per-connection 
queues, each queue for receiving packets from. r 
a respective source and temporarily storing 
j - received packets before routing to a particular . 
destination; 

servicing packets from each of said plurality of 
r per-eonnection queues.at guaranteed pre-allo- 
■ v . cated rates;. , < .\ ' 
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- sensing .a presence , or absen.ce of packets in . 
r. .-queues,- .said .absence pi. packets in queues : „ 

indicating ayailability,of excess bandwidth; and, , 

for redistributing excess bandwidth. upon sens- ; 

ing; of queues absent? said packets, said state 
; dependent scheduling means* servicing those %t 

queuesiin accordance with a.state variable cor-, 
^responding to a. performance property , of said 

queues, while preserving delay and isolation . 
, -properties for routing of said packets, -i 
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(54) Fair queuing system with adaptive bandwidth redistribution 



(57) Apparatus for routing packets in a communfca- 
tion network comprises a plurality (100) of per^sonriec**— 

tion queues (20a--20j), each queue established for 

receiving packets-from'a respective source and tempo 1 
rarily storing received packets before routing to a paftic-_ 
ular destination; a weighted fair-queuipg scheduler (40) 
for servicing packets from each of the plurality of per- t 
connection queues at guaranteed pre-allocated rates; a 
sensing device for sensing a presence or absence of 
packets in queues, the absence of packets in queues 
indicating availability of excess bandwidth; and, a state \ 
dependent scheduler (200) for redistributing, excess 
bandwidth upon sensing of queues absent packets, the 
state dependent scheduler (200) servicing those 
queues in accordance with a state variable a^respond-^ 
ing to a performance property of the queues^wher^ijr;; 
delay and isolation properties:for routing packets" of 
respective queues in weighted fair-queuing. is«pre-,~. 
served. 
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